Classifying image features

ABSTRACT

Methods are disclosed for classifying different parts of a sample into respective classes based on an image stack that includes one or more images.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S.application Ser. No. 13/091,492, filed on Apr. 21, 2011, now U.S. Pat.No. 8,280,140 which is a continuation of U.S. application Ser. No.12/477,330, filed on Jun. 3, 2009, now U.S. Pat. No. 7,953,264, which isa continuation of U.S. application Ser. No. 11/342,272, filed on Jan.27, 2006, now U.S. Pat. No. 7,555,155, which claims priority to U.S.Provisional Patent Application No. 60/647,729 entitled “METHOD FORCLASSIFYING LABELED PATHOLOGY AND CYTOLOGY TISSUE SECTIONS” by RichardLevenson and Clifford C. Hoyt, filed on Jan. 27, 2005. The contents ofthe prior applications are incorporated herein by reference in theirentirety.

TECHNICAL FIELD

This invention relates to classifying tissue samples.

BACKGROUND

Chromogenic staining techniques have been developed empirically toimpart visual contrast to various elements within tissue samples.Staining techniques and protocols can produce mixtures of dyes indifferent tissue elements, and human observers, using microscopes andother imaging devices, have learned to distinguish these stainingpatterns as typical for particular elements. Modern targeted stainingmethods, which can be specific to chemical moieties and/or molecularstructural arrangements, can produce stained tissues in which two ormore chromogenic or fluorescent stains apparently overlap spatially. Infact, the perceived overlap can result because the multiple stains trulyare bound within a common structure in the sample, or because, due tothe method of preparation, a structure within the sample containing onestain overlaps with a second structure containing a different stain. Ineither case, it may be difficult to distinguish the presence andrelative distribution of the multiple stains and the structures to whichthey are bound, especially when the stains employed have similarspectral absorption and/or emission characteristics.

In fields such as pathology and cytology in which staining andinspection of tissue samples occurs frequently, the stained samples areoften classified according to one or more criteria by human researchersperforming visual inspection of the samples using a microscope or otherimaging device. For example, a sample can be stained with multiple dyesin order to highlight differences in particular organelles, structures,or molecular targets among cells in the sample. Samples containingdifferent types of cells can be treated with different dyes in order tovisually distinguish the number, spatial distribution, and morphology ofthe cell types. The samples can then be classified according to one ormore criteria such as the presence of different types of chemical orbiological structures therein. A wide variety of staining protocols havebeen developed in order to provide different types of classificationinformation for particular classes of samples.

As an alternative to the sometimes tedious procedure of manualinspection and classification of tissue samples, machine-vision methodscan be employed in an effort to automate the process of sampleclassification.

SUMMARY

In general, in a first aspect, the invention features a method thatincludes classifying different parts of a sample into respective classesbased on an image stack that includes one or more images. For example,the sample can be a tissue section.

Embodiments of the method can include any of the following features.

The method can further include decomposing a set of spectral images of asample into an unmixed image set, where each member of the unmixed imageset corresponds to a spectral contribution from a different component inthe sample, and where the images in the image stack used forclassification include one or more of the unmixed images. For example,the images in the image stack used for classification can include someor all of the unmixed images.

Classifying can include: (i) positioning a sampling window within theimage stack to select a portion of the image stack for classification,where the selected portion includes multiple pixels; (ii) classifyingthe selected portion into one of several classes, where each of thepixels in the selected portion are provisionally classified as havingthe same class as that of the selected portion; (iii) translating thesampling window to select a second portion of the image stack forclassification and classifying the second portion into one of severalclasses, where each of the pixels in the second portion areprovisionally classified as having same class as that of the secondportion; (iv) repeating the translating and classifying for theadditional portions of the image stack until at least some of the pixelsin the image stack have been provisionally classified multiple times aspart of different portions selected by the sampling window; and (v)classifying each of at least some of the pixels that have beenprovisionally classified multiple times into one of the several classesbased on their multiple provisional classifications. The differentportions selected by the sampling window can include the same number ofpixels, and at least some of the different portions selected by thesampling window can overlap with one another. The provisionalclassifications of each pixel can be expressed as a histogram indicatingthe number of times the pixel was provisionally classified in eachclass, and the final classification of each pixel can correspond to theclass to which it was most frequently provisionally classified. Thenumber of times at least some of the pixels are provisionally classifiedcan be more than two and no larger than the number of pixels in thesampling window. For example, the number of times at least some of thepixels are provisionally classified can equal the number of pixels inthe sampling window. Additionally, the image stack can include only oneimage.

The image stack can include more than three spectral images, and theclassification can include classifying different regions of the imagestack into respective classes based on the set of spectral images, whereeach region includes multiple pixels so that each classificationinvolves both spectral and spatial information.

The method can further include generating a composite image based on aset of spectral images of the sample, where the spatial intensities oftwo or more different spectral images in the set are weighteddifferently and combined to produce the composite image, and where theone or more images in the image stack include the composite image. Forexample, the set of spectral images can include n images, and the one ormore images in the image stack used for classification can include fewerthan n images. The composite image can be generated by weighting thespatial intensities of the two or more different spectral images in theset according to a function that changes monotonically with a spectralwavelength. The weighting function can be a ramp function that varieslinearly with spectral wavelength. Alternatively, the spatialintensities of the two or more different spectral images can be weightedaccording to a function that changes non-monotonically with a spectralwavelength. For example, the weighting function can include a firstportion that changes monotonically with the spectral wavelength and asecond portion that changes monotonically with the spectral wavelength,where the slopes of the first and second portions of the weightingfunction have opposite signs (e.g., the weighting function can be aGaussian function). The weighting function can be selected to enhance acontrast between features contributed to the composite image from thetwo or more different spectral images. Further, the one or more imagesin the image stack can include two or more composite images.

In any of the methods, a neural network can be used for the classifying.Classifying different regions of the sample into the different classescan include identifying selected regions of the image stack thatcorrespond to each of the individual classes, training the neuralnetwork to recognize the classes based on the selected regions, andapplying the trained neural network to the additional regions of theimage stack. The input into the neural network can be a feature vectorhaving one or more elements based on calculating at least one spatialgray level dependency matrix. Alternatively, or in addition, the inputinto the neural network can be a feature vector having one or moreelements based on calculating a two-dimensional Fourier transform.

In certain embodiments, the one or more images in the image stack caninclude one or more spectral images. The spectral images can be imagesof sample emission according to different spectral indices for theemission, for example. Alternatively, the spectral images can be imagesof sample emission according to different spectral indices forillumination of the sample causing the emission. Further, the inputinformation for the classifying can include both spectral and spatialinformation. The sample can include components having differentabsorption and emission spectra. In addition, a number of classes intowhich regions of the sample are classified can be equal to a number ofdistinct spectral contributors in the sample. For example, the distinctspectral contributors can be chemical dyes or fluorescent labels.

In certain embodiments, the image stack can include an RGB image.

Also, in certain embodiments, one can further include generating anoutput image showing the classified regions of the sample. Additionally,any of the methods can further include obtaining the one or more imagesin the image stack. The images can be obtained, for example, bymeasuring light transmitted through, or reflected from, the sample. Theimages can also be obtained by measuring fluorescence emission from thesample.

In general, in another aspect, the invention features a method thatincludes: (i) positioning a sampling window within an image stack toselect a portion of the image stack for classification, where the imagestack includes one or more images and the selected portion includesmultiple pixels; (ii) classifying the selected portion into one ofseveral classes, where each of the pixels in the selected portion areprovisionally classified as having the same class as that of theselected portion; (iii) translating the sampling window to select asecond portion of the image stack for classification and classifying thesecond portion into one of several classes, where each of the pixels inthe second portion are provisionally classified as having same class asthat of the second portion; (iv) repeating the translating andclassifying for the additional portions of the image stack until atleast some of the pixels in the image stack have been provisionallyclassified multiple times as part of different portions selected by thesampling window; and (v) classifying each of at least some of the pixelsthat have been provisionally classified multiple times into one of theseveral classes based on their multiple provisional classifications.

Embodiments of the method can include any of the foregoing aspects orfeatures of other methods that are suitable for this method.

In general, in another aspect, the invention features apparatus thatincludes a computer readable medium storing a program that causes aprocessor to carry out any of the foregoing methods.

In general, in another aspect, the invention features apparatus thatincludes a means for obtaining one or more images of a sample, and anelectronic processor for analyzing an image stack based on the obtainedimages and configured to classify different parts of the sample intorespective classes based on the image stack as set forth in any of theforegoing methods.

Embodiments of the apparatus can include any of the following features.

The means for obtaining the one or more images of the sample can includemeans for obtaining spectrally-resolved emission images from the sample.The means for obtaining the one or more images of the sample can includemeans for obtaining images from the sample corresponding to differentspectral illuminations of the sample.

In general, in another aspect, the invention features apparatus thatincludes an optical system for obtaining one or more spectral images ofa sample, and an electronic processor for analyzing an image stack basedon the obtained spectral images and configured to classify differentparts of the sample into respective classes based on the image stack asset forth in any of the foregoing methods.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. In case of conflict betweendocuments incorporated herein by reference and the presentspecification, the present specification will control.

The details of one or more embodiments of the invention are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages of the invention will be apparent from thedescription and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram of a system for acquiring spectral imagesof a sample, and for classifying the sample.

FIG. 2 is a flow chart showing steps involved in classifying a sample.

FIG. 3 is a flow chart showing steps involved in training a neuralnetwork to perform sample classification.

FIG. 4 is a schematic diagram showing a region of interest selected fora particular class.

FIG. 5 is a schematic diagram showing a partitioning of a spatialFourier transform of a sample image in frequency space into a set ofsmaller regions.

FIG. 6 is a flow chart showing steps involved in optimizing a trainedneural network.

FIG. 7 is a flow chart showing steps involved in classifying a samplewith a trained neural network.

FIG. 8 is a schematic diagram showing a region of a sample imagesselected for classification.

FIG. 9 shows a calculation of a spatial gray level dependency matrix.

FIGS. 10A-10I show an example of a classification technique disclosedherein being applied to data for a real sample.

FIG. 11 is a schematic diagram of a portion of a neural network.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION Overview

The methods and systems disclosed herein can be used to classify a widevariety of biological and other samples according to spectral and/orstructural features appearing on images of the samples. Theclassification methods include at least some steps that are performed inan automated manner using various machine-vision algorithms andtechniques. A set of images of a sample is acquired, and can betransformed prior to submission to an automated classifier.Transformation of the image set can include mathematical transformationssuch as conversion from intensities to optical densities, spectralunmixing operations, calculation of composite images, and forming aclassification data set that may include only a subset of availablesample images. The classification data set is then submitted to amachine-based classifier, which can be a neural network or another typeof classifier. Image pixels can be classified multiple times, and afinal classification performed based on the distribution of the multipleclassifications for each pixel. Images illustratingdifferently-classified regions of the sample can be displayed for asystem operator. Classification information can also be used to as aninput to direct automated processes such as laser-capturemicrodissection, or in other image-guided procedures.

The classification methods are mathematical and so are general in scope,and can be applied wherever classification is desired, regardless of theapparatus or method used to obtain the images, or the nature of thesample. The classification methods can be used to classify a widevariety of samples, including samples stained with one or more absorbingstains, and samples that include fluorescent labels. Fluorescent labelscan include chemical labels that are introduced into a sample from anexternal source; alternatively, the labels can be intrinsic to thesample (e.g., endogenous autofluorescence or fluorescent proteins suchas green fluorescent protein and red fluorescent protein). Theclassification methods can also be used to classify samples containingvarious luminescent species and structures. The images may be obtainedin the visible, infrared, or ultraviolet range. The classificationmethods are not limited to use with images of sample absorption orsample emission, but can also be used to classify images that utilize awide variety of measurement or contrast mechanisms to visualize asample, including but not limited to polarized light, samplebirefringence, elastic or inelastic light scattering, or fluorescencelifetime. The classification methods can also be used to classifysamples that are imaged with non-optical means such as x-raytransmission or scatter, magnetic resonance, neutron scatter, orpositron emission. In short, the classification methods may be used toclassify sample regions in any setting where classification of an imageis desired. Moreover, the images may be images other than microscopicimages. For example, the images can be macroscopic images captured inremote sensing applications. Such images can be detected optically orthrough other means, as described above.

As used herein, the term “classifying” refers to identifying differentregions of an image of a sample that share a set of commoncharacteristics, wherein at least some of the steps in the procedure areperformed in an automated fashion by electronic components. The set ofcommon characteristics can include signal strength, shape, spectral andtextural features, for example. The identification of such regions in asample image effectively identifies the corresponding regions in thesample as sharing a set of common features, and more generally that thesample region is of a specific known state or type based on itsexpression of these features. At least some of the steps in theclassification procedure are performed in automated fashion byelectronic components. For example, in many embodiments, steps thatinclude spectral unmixing of images, generating composite images, andclassifying regions of images into one or more classes are performed byelectronic components. However, some operator intervention may occur inother steps. In particular, in some embodiments, steps such as theselection of reference regions corresponding to various classes fortraining a machine-based classifier may be performed manually by asystem operator.

In certain embodiments, spectral images of a sample are “unmixed” intoimages that each correspond to a spectral index of a respectiveconstituent of the sample. These unmixed images can then by processed bythe classifier. The use of the unmixed images as the input into theclassifier may improve the efficiency and/or accuracy of theclassification.

In certain embodiments, one or more composite images can be generatedfrom spectral images, prior to classification. As explained in moredetail later, composite images generally include “flattened” spectralinformation; that is, composite images contain spectral information thatis encoded as variations in a spatial intensity image of a sample. Theuse of the composite image as an input into the classifier may improvethe efficiency and/or accuracy of the classification.

In certain embodiments, the classification may involve the use of asampling window to initially classify the pixels in the window, followedby subsequent translations of the sampling window to make furtherclassifications. The translations are smaller than a dimension of thewindow, so that pixels are classified multiple times. A finalclassification of each pixel is then based on the statisticaldistribution of the initial classifications. The technique enables theuse of sampling window large enough to recognize spatial featuresindicative of a specific class, while still providing fine resolutionbecause of the smaller-scale translations.

In general, the classification methods disclosed herein can be used toclassify features in spectral image sets, including color (RGB) imagesof a sample; or the methods can be used for sample classification wheresample images contain no spectral information (i.e., gray scale ormonochrome images).

Apparatus for Obtaining Images and Subsequent Classification

FIG. 1 is a schematic diagram showing a system 100 for acquiringmultiple spectrally resolved images of a sample, and for classifying thesample. A light source 102 provides light 122 to light conditioningoptics 104. Light 122 can be incoherent light, such as light generatedfrom a filament source for example, or light 122 can be coherent light,such as light generated by a laser. Light 122 can be eithercontinuous-wave (CW) or time-gated (i.e., pulsed) light. Further, light122 can be provided in a selected portion of the electromagneticspectrum. For example, light 122 can have a central wavelength and/or adistribution of wavelengths that falls within the ultraviolet, visible,infrared, or other regions of the spectrum.

Light conditioning optics 104 can be configured to transform light 122in a number of ways. For example, light conditioning optics 104 canspectrally filter light 122 to provide output light in a selectedwavelength region of the spectrum. Alternatively, or in addition, lightconditioning optics can adjust the spatial distribution of light 122 andthe temporal properties of light 122. Incident light 124 is generatedfrom light 122 by the action of the elements of light conditioningoptics 104.

Incident light 124 is directed to be incident on sample 108 mounted onillumination stage 106. Stage 106 can provide means to secure sample108, such as mounting clips or other fastening devices. Alternatively,stage 106 can include a movable track or belt on which a plurality ofsamples 108 are affixed. A driver mechanism can be configured to movethe track in order to successively translate the plurality of samples,one at a time, through an illumination region on stage 106, whereonincident light 124 impinges. Stage 106 can further include translationaxes and mechanisms for translating sample 108 relative to a fixedposition of illumination stage 106. The translation mechanisms can bemanually operated (e.g., threaded rods) or can be automatically movablevia electrical actuation (e.g., motorized drivers, piezoelectricactuators).

In response to incident light 124, emitted light 126 emerges from sample108. Emitted light 126 can be generated in a number of ways. Forexample, in some embodiments, emitted light 126 corresponds to a portionof incident light 124 transmitted through sample 108. In otherembodiments, emitted light 126 corresponds to a portion of incidentlight 124 reflected from sample 108. In yet further embodiments,incident light 124 can be absorbed by sample 108, and emitted light 126corresponds to fluorescence emission from sample 108 in response toincident light 124. In still further embodiments, sample 108 can beluminescent, and may produce emitted light 126 even in the absence ofincident light 124. In some embodiments, emitted light 126 can includelight produced via two or more of the foregoing mechanisms.

In many embodiments, sample 108 is a biological sample such as a tissueslice (e.g., a sample used for pathology, or a cell suspension or smear,as in cytology studies), or living or fixed cells in tissue culture. Insome embodiments, sample 108 can be an animal (e.g., a mouse),individual bacteria or other microorganisms, bacterial or othercolonies, embryos, oocytes, plants, including seeds or grains, or sample108 can be a non-biological entity.

Light collecting optics 110 are positioned to received emitted light 126from sample 108. Light collecting optics 110 can be configured tocollimate emitted light 126 when light 126 is divergent, for example.Light collecting optics 110 can also be configured to spectrally filteremitted light 126. Filtering operations can be useful, for example, inorder to isolate a portion of emitted light 126 arising via one of themechanisms discussed above from light arising via other processes.Further, light collecting optics 110 can be configured to modify thespatial and/or temporal properties of emitted light 126 for particularpurposes in embodiments. Light collecting optics 110 transform emittedlight 126 into output light 128 which is incident on detector 112.

Detector 112 includes one or more elements such as CCD sensorsconfigured to detect output light 128. In embodiments, detector 112 canbe configured to measure the spatial and/or temporal and/or spectralproperties of light 128. Detector 112 generates an electrical signalthat corresponds to output light 128, and is communicated via electricalcommunication line 130 to electronic control system 114.

Electronic control system 114 includes a processor 116, a display device118, and a user interface 120. In addition to receiving signalscorresponding to output light 128 detected by detector 112, controlsystem 114 sends electrical signals to detector 112 to adjust variousproperties of detector 112. For example, if detector 112 includes a CCDsensor, control system 114 can send electrical signals to detector 112to control the exposure time, active area, gain settings, and otherproperties of the CCD sensor.

Electronic control system 114 also communicates with light source 102,light conditioning optics 104, illumination stage 106, and lightcollecting optics 110 via electrical communication lines 132, 134, 136,and 138, respectively. Control system 114 provides electrical signals toeach of these elements of system 100 to adjust various properties of theelements. For example, electrical signals provided to light source 102can be used to adjust the intensity, wavelength, repetition rate, orother properties of light 122. Signals provided to light conditioningoptics 104 and light collecting optics 110 can include signals forconfiguring properties of devices that adjust the spatial properties oflight (e.g., spatial light modulators) and for configuring spectralfiltering devices, for example. Signals provided to illumination stage106 can provide for positioning of sample 108 relative to stage 106and/or for moving samples into position for illumination on stage 106,for example.

Control system 114 includes a user interface 120 for displaying systemproperties and parameters, and for displaying captured images of sample108. User interface 120 is provided in order to facilitate operatorinteraction with, and control over, system 100. Processor 116 includes astorage device for storing image data captured using detector 112, andalso includes computer software that embodies instructions to processor116 that cause processor 116 to carry out control functions, such asthose discussed above for example. Further, the software instructionscause processor 116 to mathematically manipulate the images captured bydetector 112 and to carry out the steps of classifying sample 108according to either or both of the original and the manipulated images.The classification steps are described in more detail subsequently.

In many embodiments, system 100 is configured to acquire multiplespectral images of sample 108. The multiple spectral images maycorrespond to illumination of sample 108 at a variety of selectedwavelengths of light, and detecting an intensity of light eithertransmitted through or reflected by sample 108. Alternatively, themultiple spectral images may correspond to illumination of sample 108with light having similar spectral properties, and collecting multipleimages of sample 108, each image corresponding to a different wavelengthof emitted light 126. Spectral filtering elements in light conditioningoptics 104 and light collecting optics 110 are generally used to obtainthe spectrally resolved data.

In some embodiments, images of sample 108 can be collected in sequence,with adjustments to the configuration of optical components (e.g.,optical filters) between successive captured images. In otherembodiments, multiple images can be captured simultaneously usingdetection systems configured to detect multiple sample views. Forexample, detection systems can be configured to project different viewsof the sample corresponding to different illumination or emissionwavelengths onto a detector such as a CCD camera, and the multiple viewscan be captured simultaneously.

In some embodiments, light conditioning optics 104 include an adjustablespectral filter element such as a filter wheel or a liquid crystalspectral filter. The filter element can be configured to provide forillumination of sample 108 using different light wavelength bands. Lightsource 102 can provide light 122 having a broad distribution of spectralwavelength components. A selected region of this broad wavelengthdistribution is allowed to pass as incident light 124 by the filterelement in light conditioning optics 104, and directed to be incident onsample 108. An image of light 126 transmitted through sample 108 isrecorded by detector 112. Subsequently, the wavelength of the filterpass-band in light conditioning optics 104 is changed to provideincident light 124 having a different wavelength, and an image of light126 transmitted through sample 108 (and corresponding to the newwavelength of incident light 124) is recorded. A similar set ofspectrally-resolved images can also be recorded by employing a lightsource 102 having multiple source elements generating light of differentwavelengths, and alternately turning the different source elements onand off to provide incident light 124 having different wavelengths.

As discussed previously, the emitted light 126 from sample 108 can alsocorrespond to incident light 124 that is reflected from sample 108.Further, emitted light 126 can correspond to fluorescence emission fromsample 108 if the sample includes fluorescent chemical structures. Forsome samples, emitted light 126 can include contributions from multiplesources (i.e., transmission and fluorescence) and the spectral filteringelements in light conditioning optics 110 can be used to separate thesesignal contributions.

In general, both light conditioning optics 104 and light collectingoptics 110 include configurable spectral filter elements. Therefore,spectral resolution can be provided either on the excitation side ofsample 108 (e.g., via light conditioning optics 104) or on the emissionside of sample 108 (e.g., via light collecting optics 110), or both. Inany case, the result of collecting multiple, spectrally resolved imagesof sample 108 is an “image stack” where each image in the stack is atwo-dimensional image of the sample corresponding to a particularwavelength. Conceptually, the set of images can be visualized as forminga three-dimensional matrix, where two of the matrix dimensions are thespatial length and width of each of the images, and the third matrixdimension is the spectral wavelength (emission or excitation) to whichthe image corresponds. For this reason, the set of spectrally resolvedimages can be referred to as a “spectral cube” of images. As usedherein, a “pixel” in such a set of images (or image stack or spectralcube), refers to a common spatial location for each of the images.Accordingly, a pixel in a set of images includes a value associated witheach image at the spatial location corresponding to the pixel.

Other arrangements to obtain spectral images which are known in the artmay be employed, according to the requirements of the sample at hand.

While each spectral image described above typically refers to aparticular wavelength or range of wavelengths (e.g., a spectral band),more generally, each spectral image can correspond to a spectral indexthat may include one or more wavelength bands, or some more complexspectral distribution. For example, such an image can be generated byusing a spectral comb filter. Generally, the image cube will includeseveral spectral images, for example, 10 or more. However, in someembodiments, the image cube may include fewer images, for example, onlytwo or three spectral images. One such example is an red-green-blue(RGB) color image, in which each pixel includes a value associated withthe strength of each of the red, green, and blue colors. Suchinformation may be displayed as a single color image, rather than as aset of separate images; however, the information content is the same asthat in the set of images, and therefore we use the expression “spectralimages” to refer to both cases.

In certain embodiments, images used for classification may also includefalse-color images, and also monochrome or gray scale images.

Following acquisition of one or more images, sample 108 is classified bysystem 100 according to the shape, intensity, spectral and/or texturalfeatures of the individual image(s). In practice, in some embodiments,images are recorded for multiple samples first, and the classificationof the samples is deferred to a later time for expediency.

Not all of the images of a spectral cube need be analyzed in order toaccurately classify the sample to which the cube corresponds. In someembodiments, a classification of sufficiently high accuracy is achievedby examining only a subset of the spectral cube images. Further, in someembodiments, the spectrally resolved images may be spectrally unmixed(i.e., decomposed into a set of images corresponding to a set ofspectral eigenstates) before analysis. Some embodiments includeadditional steps wherein one or more composite images are generated viamathematical combination of multiple images selected from the spectralcube and/or the set of spectrally unmixed images. Classification of asample can be performed based on the composite images, in addition to orexclusive of the spectral cube images and the spectrally unmixed images.

Spectral Unmixing

FIG. 2 is a flow chart 200 showing steps involved in classifying asample. Step 202 includes acquiring a set of one or more images (e.g., aspectral cube) of a sample, as discussed above. Step 204, which isoptional, includes spectrally unmixing some or all of the images in thespectral cube to generate an unmixed set of images (i.e., an “unmixedspectral cube”). Spectral unmixing is a technique that quantitativelyseparates contributions in an image that arise from spectrally differentsources. For example, a sample may contain three different types ofstructures, each labeled with a different dye. The three different dyesmay each have different absorption spectra. Typically, the individualabsorption spectra of the dyes are known before they are used, or theycan be measured. Images of the specimen under illumination will contain,in the most general case, spectral contributions from each of the threedyes. A similar situation arises, for example, in samples containingmultiple different fluorescence labels, each of which contribute tomeasured fluorescence emissions.

Spectral unmixing decomposes one or more images that includecontributions from multiple spectral sources into a set of componentimages (the “unmixed images”) that correspond to contributions from eachof the spectral entities within the sample. Thus, if the sample includesthree different dyes, each specific to a particular structural entity,then an image of the sample can be separated into three unmixed images,each unmixed image reflecting contributions principally from only one ofthe dyes.

The unmixing procedure essentially corresponds to decomposing an imageinto a set of spectral eigenstates. In many embodiments, the eigenstatesare known beforehand, as discussed above. In other embodiments, theeigenstates can sometimes be determined using techniques such asprincipal component analysis. In either case, once the eigenstates havebeen identified, an image can be decomposed by calculating a set ofvalues, usually as a coefficient matrix, that corresponds to therelative weighting of each of the eigenstates in the overall image. Thecontributions of each of the individual eigenstates can then beseparated out to yield the unmixed image set.

As an example, a series of two dimensional images having x and ycoordinates can be measured for a sample by illuminating the sample at aset of different excitation wavelengths λ_(k). As described above, thetwo dimensional images can be combined to form a three-dimensional imagecube I(x,y,k) where the first two indices of the image cube representcoordinate directions, and the third index is a spectral indexcorresponding to the wavelength of the illumination light. Assuming, forthe sake of simplicity, that each of the images of the sample containsspectral contributions from two different spectral sources F(λ_(k)) andG(λ_(k)), then the values in the three-dimensional image cube I(x,y,k)may be given byS(x,y,k)=a(x,y)·F(λ_(k))+b(x,y)·G(λ_(k))  (1)where λ_(k) is used to denote a given wavelength (or wavelength band).The functions a(x,y) and b(x,y) describe the spatial abundance of thespectral contributions from the two different spectral sources in thesample.

According to Equation (1), the net signal any position in thethree-dimensional image cube (i.e., at any two-dimensional pixelcoordinate, and at a particular illumination wavelength) is the sum oftwo contributions, weighted by the relative abundance of each. This canbe expressed asI(λ_(k))=aF(λ_(k))+bG(λ₁)  (2)

The functions F and G can be termed the “spectral eigenstates” for thesystem because they correspond to the pure spectra for the spectralsources in the sample, which are combined in varying proportions toproduce the measured spectral images of the sample. Thus, the samplespectrum is a weighted superposition corresponding to separatecontributions from the two spectral sources.

If the spectra F(λ_(k)) and G(λ_(k)) are known (or can be deduced), thenEquation (2) can be inverted to solve for a and b, provided thatspectrum I includes at least two elements (i.e., provided that one hasdata for at least two wavelengths λ_(k)). Equation (2) can be rewrittenin matrix form as I=EA, so thatA=E ⁻¹ I  (3)where A is a column vector with components a and b, and E is a matrixwhose columns are the spectral eigenstates, namely [F G].

Using Equation (3), measured spectral images of a sample can be used tocalculate contributions to the images arising purely from source F andpurely from source G at particular pixel locations. The process can berepeated for each pixel location on a selected image (i.e., throughoutthe range of values x and y in I) to produce an image of the sample thatincludes contributions only from source F, and another image of thesample that includes contributions only from source G.

In the above discussion, the number of spectral sources is two (i.e., Fand G). In general, however, unmixing techniques are not restricted toany particular number of sources. For example, a sample can generallycontain m different spectral sources. If the number of wavelengths atwhich data is collected is n—that is, k=1 . . . n—then matrix E is ann×m matrix instead of an n×2 matrix, as in the above discussion. Theunmixing algorithm can then be employed in the same manner as describedabove to isolate specific contributions at each pixel location in animage from each of the m spectral eigenstates.

One factor which can limit the ability of the algorithm to distinguishbetween contributions from different spectral eigenstates is the degreeof spectral distinction between the eigenstates. The correlation betweentwo spectra, such as two spectral eigenstates I₁ and I₂, can bedescribed by a spectral angle θ where

$\begin{matrix}{\theta = {\cos^{- 1}\left\lbrack \frac{I_{1} \cdot I_{2}}{{I_{1}}{I_{2}}} \right\rbrack}} & (4)\end{matrix}$

Sets of spectra for which θ is small for two members are not as easilyseparated into their components. Physically, the reason for this iseasily understood: if two spectra are only marginally different, it isharder to determine the relative abundance of each.

A number of techniques can be used to measure or estimate the purespectra of the spectral sources F and G (and other spectral sources,where the sample includes more than two). In general, any method thatyields spectral eigenstates of sufficient accuracy can be used. Somesamples can contain spectral sources such as dyes, fluorescence labels,or other chemical moieties for which there are known spectra availablein published reference materials. Alternatively, it may be possible todirectly measure the spectra of source components using one or moremeasurement systems. In some samples, a particular region of the samplemay be known to include only one particular spectral source, and thespectrum of that source can be extracted from measurements taken on onlythe identified region of the sample.

Various data analysis techniques can also be used for determiningcomponent spectra for spectral unmixing, such as principal componentanalysis (PCA), which identifies the most orthogonal spectraleigenvectors from an image cube and yields score images showing theweighting of each eigenvector throughout the image. This may be done incombination with other mathematical processing, and there are otherknown techniques for identifying low-dimensionality spectral vectors,such as projection pursuit, a technique described, for example, in L.Jimenez and D. Landgrebe, “Hyperspectral Data Analysis and FeatureReduction Via Projection Pursuit”, IEEE Transactions on Geoscience andRemote Sensing, Vol. 37, No. 6, pp. 2653-2667, November 1999, the entirecontents of which are incorporated herein by reference. Other techniquesinclude independent component analysis (ICA) and end-member detectionalgorithms, for example.

These techniques are typically not well-suited to the applications inthe life sciences. For example, some techniques are optimized forspectral imaging data sets that contain spectra with dense spectralshapes and well-defined narrow peaks. In some techniques the spectralranges are large compared to the individual spectral features and peaksthat are used for analysis. The presence of peaks, or the ratio of peaksmay be then used to classify “end-members” to be separated.Unfortunately, the components in biological samples typically do nothave such well-defined, narrow peaks.

Some of these techniques generate images related to spectra that arepresent in a pure form somewhere within the original image cube. In manycases in the life sciences, signal spectra present in the image cube aremixtures of components. If the component of interest is not in a pureform somewhere in the original image cube, then it is unlikely thatthese techniques will generate an image that accurately represents theabundance of the component of interest.

There are some techniques, sometimes called “convex-hull” algorithms,that estimate what the true end-members are even if they do not exist ina pure form in the image, but the effectiveness is dependent on howclose signal spectra in the image cube are to the end-members.

One technique that can be used to extract spectral eigenstates (orrepresentations thereof) without a priori knowledge of all of theeigenstates involves considering the signal spectrum I(λ_(k)) for agiven pixel, and subtracting from it the maximum amount of a firstspectral source F(λ_(k)) while leaving the remaining signal that ispositive definite in all spectral channels. That is, one defines aso-called “remainder spectrum” U_(a)(λ_(k)) for each pixel asU _(a)(λ_(k))=I(λ_(k))−aF(λ_(k))  (5)and then selects the largest value of the parameter a consistent withU_(a)(λ_(k)) having a non-negative value in every spectral channel. Theresulting spectrum U_(a)(λ_(k)) is then used as the signal spectrum,expunged of contributions due to first spectral source F. One may alsomake the determination of parameter a based not on strict non-negativecriterion listed above, but on some related criteria that incorporates asmall negative distribution, to account for considerations such as shotnoise or detector noise in a measurement system. Additional examples ofoptimization criteria for removing the maximal amount of spectral sourceF include using different error functions.

Alternatively, one may seek to extract a contribution to a measuredspectrum that is due to second spectral source G. In analogy withEquation (5), the remainder spectrum can be calculated for each pixel asU _(b)(λ_(k))=I(λ_(k))−bG(λ_(k))  (6)where one selects the largest value of the parameter b consistent withU_(b)(λ_(k)) having a non-negative value in every spectral channel.

The remainder technique can be expanded to cases where the spectra forone or more additional components of the sample are known, and one wantsto remove their contributions to the signal. In such cases, theremainder spectrum is written to subtract a contribution of each suchcomponent from the observed signal based on the additional spectra andconsistent with a positive remainder in each spectral channel.

Additional spectral unmixing techniques are described in PCT PatentPublication No. WO2005/040769 entitled “SPECTRAL IMAGING OF BIOLOGICALSAMPLES” by Richard Levenson et al., the contents of which areincorporated herein by reference.

In order for the spectral unmixing techniques disclosed herein toeffectively separate contributions in sample images that are due todifferent spectral eigenstates, Equation (1) should be at leastapproximately correct. That is, the measured spectral data should beapproximately described as a linear superposition of weightedeigenstates. This approximation holds for many samples and spectralmeasurement techniques, especially darkfield measurement techniques. Forexample, sample images arising from fluorescent or luminescent chemicallabels within the sample typically satisfy the linearity assumption. Insome cases however, such as for some brightfield measurement techniques,the linearity approximation may not be satisfied. For example, whenimages are captured that arise from illumination light that istransmitted through a sample that includes light-absorbing components,the linearity assumption in Equation (1) may not be correct. Instead,the intensity of the measured light may be reduced with an exponentialdependence on the concentration of the light-absorbing components. Insuch cases, transformation of the images may first be necessary beforeunmixing techniques can be used. As an example, for sample imagesmeasured in a transmission mode, the measured image intensities can betransformed into optical densities (e.g., by applying a logarithmicfunction) in order to apply linear unmixing techniques. Optical densitytechniques are further described, for example, in U.S. application Ser.No. 10/226,592 (Publication No. US 2003/0081204 A1) entitled “SPECTRALIMAGING” by Paul J. Cronin and Peter J. Miller, filed Aug. 23, 2002, theentire contents of which are incorporated herein by reference.

Spectral unmixing operations (e.g., matrix inversion techniques andremainder techniques) and image data transformation operations (e.g.,converting measured image intensities to optical densities, whereappropriate) can be performed by electronic control system 114 viaprocessor 116, for example. These operations can include manualintervention and configuration steps performed by a system operator, orsystem 100 can be configured to perform these operations in an automatedmanner.

Composite Images

Application of the unmixing techniques discussed above provides a set ofunmixed images from a multi-spectral data set. Returning now to FIG. 2,in a second optional step in flow chart 200, step 206 includesgenerating one or more composite images using the spectral cube imagesand/or unmixed spectral cube images. Composite images are generated as ameans to “flatten” or compress spectral information into atwo-dimensional grayscale image. In other words, in terms of a 3Dspectral matrix of image data, generating a composite image correspondsroughly to compressing or packing the information from two or morelayers into a single layer. Since both spectral cube and unmixedspectral cube image data can be used, the technique can conceptuallyinclude packing multiple layers from different spectral cubes into asingle layer.

As an example, consider a 3D spectral cube of images, where each imagehas width x, height y, and an index k that corresponds to a wavelengthλ_(k). If there are a total of N different images in the cube (i.e.,data recorded at N different wavelengths) then the spectral cube I canbe represented, as described previously, as a matrix I(x,y,k).Compressing spectral information from two or more images in the spectralcube to create a composite image C is equivalent to adding the imagelayers together. In some embodiments, prior to adding the layerstogether, each layer is scaled according to a weighting function ƒ(k).The spectral compression operation is then performed according to

$\begin{matrix}{{C\left( {x,y} \right)} = {\sum\limits_{k = m}^{n}{{f(k)} \cdot {I\left( {x,y,k} \right)}}}} & (7)\end{matrix}$which yields composite image C(x,y) from layers m through n of thespectral image cube. The weighting function ƒ(k) is generally chosen toemphasize different spectral features in the composite image; that is,to create contrast between features arising from the different layers ofthe spectral cube that contribute to the overall intensity distributionin the composite image.

A wide variety of weighting functions can be chosen in order to producethe desired contrast. In general, in some embodiments, a monotonicallyincreasing or decreasing function is chosen for ƒ(k), such as a linearramp function or a sigmoidal function. In other embodiments, ƒ(i) can bea dual ramp function (i.e., decreasing to a point and then increasing,or increasing to a point and then decreasing) or another function, suchas one or more Gaussian functions. The weight function can generally beselected as desired, and can be applied to a batch series of samples, orcan be selected individually for each sample prior to classification.System 100 can include a storage medium to store weighting functions forparticular types of samples, so that a weighting function appropriatefor a sample undergoing classification can be recalled as needed.

Step 208 includes selecting a set of images to be classified. Ingeneral, any or all of the images from the spectral image cube, theunmixed spectral image cube (if calculated), and the composite images(if calculated) can be selected for classification analysis. In someembodiments, for example, classification of a sample to a high degree ofaccuracy can be achieved using a composite image and a small subset ofeither spectral cube images or unmixed spectral cube images. This hasthe advantage that the overall set of data upon which a classificationalgorithm operates is greatly reduced, increasing the speed with whichthe classification of the sample is complete.

In some other embodiments, images from the unmixed spectral cube can beused for sample classification. The images can be delivered to aclassification algorithm, and may be accompanied (although not always)by one or more composite images.

In some embodiments, more than three spectral images can be used forclassification of a sample. The images can be taken from either thespectral image cube or, if calculated, an unmixed spectral image cube.This technique can be particularly advantageous when the sample includesmore than three distinct spectral contributors. For example, the samplecan contain four different stains or dyes, or four different fluorescentlabels.

In other embodiments, color RGB images or single plane images can beused for classification of a sample. Single plane images may be narrowband or panchromatic.

Classification

In general, the classifier is a mechanism or rule-set to assign a sampleto one of several output classes, and it can be any linear or nonlinearclassifier. Linear classifiers include least-squares distance,Mahalanobis distance, and others. These may be used, but the classifieris preferably a machine-learning algorithm such as a neural network,genetic algorithm, or support vector machine. However, a neural networkis often preferred, and will be used as the example throughout thesubsequent discussion.

The neural network is generally applied to one or more areas, each ofwhich typically corresponds to several pixels (e.g., a 2×2 set ofpixels, or a 16×16 set of pixels, etc.) in the image stack (which, asdescribed above, may include one or more images). When there is morethan one image in the image stack, each pixel will include a valueassociated for each of the images. The values for all of the pixels in agiven area being classified form the basis of the input information thatcan potentially be applied to the neural network. Because each areaincludes several pixels, the input information available to the neuralnetwork includes both spatial and spectral information when the imagestack includes a composite image and/or multiple spectral images.

The neural network has one or more input nodes, by which it receivesinformation about the region to be classified. An input is termed a“feature vector,” where each element of the feature vector correspondsto a specific input node of the neural network. The elements of thefeature vector are functions of the signal values at one or more pixelsin the area being classified. Examples of suitable functions forproducing the feature vector are described further below.

The neural network will also have several output nodes, eachcorresponding to a class to which the area may be designated. When thefeature vector for a given area is applied to the neural network, valuesfor the output nodes correspond to the degree to which the area shouldbe assigned to a given class. Preferably, the neural network is trainedso that the output node values are binary, with only one output nodeyielding a non-zero value (and indicating the class to which the areashould be assigned) for any given feature vector.

As described in further detail below, the neural network is trained andcan be further optimized to reduce the number of input nodes necessaryfor efficient and accurate classification. In many embodiments, the useof unmixed images and/or one or more composite images can lead to areduction in the number of input nodes, and therefore greater efficiencywhen classifying the regions of an unknown sample. The topology of theneural network employed in some embodiments is bipolar, although binaryand other neural network types can also be used effectively. The networkis trained using a back propagation method, with momentum included inthe training algorithm. The activation function of the network in someembodiments is a bipolar sigmoidal function; other activation functionscan also be used.

In embodiments, the networks can commonly include 0, 1, or 2 hiddenlayers, which are the layers between a first layer having the inputnodes and the last layer having the output nodes, although additionalhidden layers are possible. Anywhere from 1-15 nodes per hidden layerand are common, though again additional nodes can be used. The inputlayer of the network uses spatial and spectral texture featuresidentified on sample images as input. The output layer includes a numberof output nodes equal to the number of identified classes N_(c).

FIG. 11 is a schematic diagram showing an example of a neural networkthat can be used in the classification methods disclosed herein. Thenetwork includes an input layer, one hidden layer, and an output layer.Inputs to the neural network are feature vectors ƒ_(m) and couplingstrengths between nodes are given by γ_(k,l) values. The outputs fromthe neural network are the classes associated with an image or imagestack.

Typical topological parameters for networks used in the processing oftissue sample images include one hidden layer with 5 nodes, a learningparameter of 0.2, and a momentum factor of 0.5. The structure of neuralnetworks are described, for example, in Christopher M. Bishop, “NeuralNetworks for Pattern Recognition”, Oxford University Press, 1995.

Referring again to FIG. 2, after selecting the set of images accordingto which the sample will be classified (the “classification image set”),step 210 includes training the classifier using images from theclassification image set.

The neural network is trained when a new type of sample is presented forclassification analysis. A system operator can be provided with a choiceto re-train the existing network for a particular sample via displaydevice 118, for example. The procedure for training the neural networkis discussed in greater detail subsequently.

After the neural network-based classifier is trained, step 212 includessubmitting the classification image set to the classifier. Theclassifier generally classifies portions of the sample according totextural and spectral features present on images of the sample in theclassification image set. The details of the steps involved in theclassification routine are presented later.

Finally, step 214 includes generating classification output for thesample. The classification output can include, for example, one or moreimages constructed to show contrast between differently classifiedregions of the sample. Alternatively, or in addition, the classificationoutput can include warning sounds or messages to indicate the presenceor absence of particular elements (i.e., stained or labeled structures)in the sample. The output can also include numeric data indicating thetypes of regions present in the sample, their relative abundance, andother numerical parameters describing the sample.

Training the Neural Network

FIG. 3 is a flow chart 300 that includes steps for training the neuralnetwork classifier. A first step 302 includes determining a number ofclasses N_(c) to search for in the image stack. In many embodiments, thenumber of classes is selected to correspond to the number of differentstates that are expected or sought within the sample. This may begreater than the number of spectral planes in the image set, or it maybe fewer. For example, a sample may be stained with three different dyesor labeled with three different fluorescent labels. In such a sample,one may seek to identify three different classes N_(c), or two, or five,according to the structure and nature of the sample. The classifier iscapable of resolving more classes N_(c) than the number of spectralplanes, based on other aspects of the sample such as signal strength,shape, and texture.

The second step 304 includes selecting at least one training region ofinterest (ROI) for every class on one of the sample images (the pixelspatial coordinates (x,y) of the ROIs for each of the classes areassumed to be the same from one image to the next). The training ROIsare known to correspond to respective classes and provide a referencefor the neural network algorithm to allow it to determine particularspectral and spatial features which are common to each of the classes inorder to assist in classification decisions. In some embodiments, forexample, the selection of ROIs occurs dynamically via interaction with asystem operator through display device 118 and user interface 120.

The third step 306 includes selecting a sub-sampling window size. Asub-sampling window is used to examine each of the selected ROIs at afiner level of detail. In many embodiments, the sub-sampling window sizeis chosen to be smaller than the mean length and width of all of theROIs, but larger than a single pixel within the ROI. The sub-samplingwindow width is also frequently chosen to have both width and lengththat are multiples of 2, because Fourier methods that operate onsub-sampled regions of the ROIs can take advantage of FFT algorithms ifthe variable space is a multiple of 2. In embodiments, typicalsub-sampling window sizes include 4×4 pixels, 8×8 pixels, 16×16 pixels,and 32×32 pixels, although a wide variety of window sizes, includingwindow sizes not listed explicitly herein, are also possible. Moreover,while the presently described embodiments presume that the data for eachimage is represented with respect to a two-dimensional grid of squares,other embodiments may include a different representation of data andcorresponding window and ROI dimensions. For example, the data mayrepresented on a hexagonal grid, or some other shape.

The next series of steps involve operations conducted on each of theidentified classes. Each of the classes is analyzed in turn. Step 308includes selecting a ROI corresponding to a currently selected class.Step 310 includes examination of the ROI by sub-sampling the ROI withthe selected sub-sampling window. FIG. 4 shows the sub-sampling processin greater detail. A chosen ROI 400 is sub-sampled by a sub-samplingwindow 402 that selects a fraction of the image pixels within ROI 400for analysis.

Returning to FIG. 3, step 312 includes calculating and storing a featurevector for each of the sub-sampled regions of the ROI. The featurevector includes as elements a set of numbers calculated from thesub-sampled pixels of the ROI. Each of the calculated feature vectorscorrespond to a feature vector that would, for a properly trained neuralnetwork, output a classification corresponding to the selected class.The elements of the feature vector generally correspond to particulartexture analysis features which provide a basis for classification ofregions within an image of the sample.

Many different numerical quantities can be calculated in order toprovide a sufficiently distinguishable description of the ROI. Forexample, in some embodiments, the feature vector corresponding to aselected ROI for a particular class can include 10 differentcalculations for each of the images in the image stack, therebyresulting in vector with 10N_(i) elements, where N_(i) is the number ofimages in the image stack. The first four of the ten calculations can betexture analysis features obtained from spatial gray level dependencymatrices (SGLDMs), which are also referred to as co-occurrence matrices.For example, such matrices are described in R. M. Haralick, K.Shanmugam, and I. Dinstein, “Textural features for imageclassification”, IEEE Trans. Syst., Man, Cybern., vol. SMC-3, pp.610-621, 1973. A SGLDM is a spatial histogram of an image (or a portionthereof) that quantifies a distribution of gray scale values within theimage. SGLDMs can be calculated, for example, from an estimate of thesecond-order joint conditional probability densities, s_(θ)(i,j|d,θ).Each value of this conditional probability density represents theprobability of a pixel having a gray level value i being d pixels awayfrom a pixel having a gray level value j in a direction described by θ.If an image includes N_(g) gray levels, then an N_(g)×N_(g) matrixs_(θ)(i,j|d,θ) can be created. Optionally, the matrix can be summed overa set of directions θ for a selected distance d. For example, in someembodiments, a single direction θ=0° can be selected. In otherembodiments, for example, four directions can be employed: θ=0°, 45°,90°, and 135°. In general, any number of directions can be selected foranalysis of the texture features in a particular ROI.

In some embodiments, the distance d is fixed at a particular value foranalysis. For example, the distance d can be fixed at a value of 1pixel. In other embodiments, a range of distances can be used, dependingupon the nature of the specific texture features. In general, thedistance d and the direction θ can be regarded as parameters that areadjusted in order to ensure higher accuracy classification performancefrom the neural network.

With four directions θ and a single fixed distance d of one pixel, forexample, a SGLDM

$\begin{matrix}{E = {\sum\limits_{i = 0}^{N_{g} - 1}{\sum\limits_{j = 0}^{N_{g} - 1}\left\lbrack {s_{\theta}\left( {i,\left. j \middle| d \right.} \right)} \right\rbrack^{2}}}} & \; \\{S = {\sum\limits_{i = 0}^{N_{g} - 1}{\sum\limits_{j = 0}^{N_{g} - 1}{{s_{\theta}\left( {i,\left. j \middle| d \right.} \right)}{\log\left\lbrack {s_{\theta}\left( {i,\left. j \middle| d \right.} \right)} \right\rbrack}}}}} & \; \\{H = {\sum\limits_{i = 0}^{N_{g} - 1}{\sum\limits_{j = 0}^{N_{g} - 1}{\frac{1}{1 + \left( {i - j} \right)^{2}}{s_{\theta}\left( {i,\left. j \middle| d \right.} \right)}}}}} & (8) \\{R = {\sum\limits_{i = 0}^{N_{g} - 1}{\sum\limits_{j = 0}^{N_{g} - 1}{\left( {i - j} \right)^{2}{s_{\theta}\left( {i,\left. j \middle| d \right.} \right)}}}}} & (9)\end{matrix}$can be computed as a sum of co-occurrence matrices over the fourdirections in each ROI. Textural features can then be calculated fromeach SGLDM. For example, four different textural features that can becalculated from each SGLDM include energy (E), entropy (S), local (10)homogeneity (H), and inertia (R). The inertia value is also referred toas “contrast”. In this example, then, four SGLDM features for the set ofangles θ can be calculated as follows for each ROI: where s_(θ)(i, j|d)corresponds to the (i,j)-th element of the SGLDM for a distance d. Thecalculated values E, S, H, and R, for each of the image slices, can thenbe stored as the first 4N elements in the feature vector correspondingto the ROI for the currently selected class.

As an example, a 2×2 region 902 of an image is shown in FIG. 9. Theregion includes 4 pixels, each of which can have an integral intensitylevel from 1 to 4 (i.e., N_(g)=4). The second-order joint conditionalprobability matrix s_(θ)(i, j|d,θ) is therefore a 4×4 matrix 904. Inorder to evaluate the numerical elements of matrix 904, particularvalues of d and θ can be selected. For example, selecting θ=0corresponds to evaluating probabilities along rows of region 902.Selecting d=1 corresponds to evaluating probabilities for elements inregion 902 that are separated by 1 unit (i.e., adjacent elements). Withthe selection of θ=0 and d=1 for region 902, the values of the elementsof probability matrix 904 are as shown in FIG. 9.

In region 902, pixel (1,1) has an intensity value of 1. Related to pixel(1,1), at a distance d=1 and angle θ=0, is pixel (1,2) with an intensityvalue of 3. Therefore, the probability value at position (3,1) in matrix904 is 1. Pixel (2,1) in region 902 has an intensity value of 1. Relatedto pixel (2,1) at a distance d=1 and angle θ=0 is pixel (2,2) with anintensity value of 2. Therefore, the probability value at position (2,1)in matrix 904 is 1. In some embodiments, the next four calculations foreach of the image slices in the ROI's feature vector can be derived fromthe magnitude of the complex 2D Fourier transform of the ROI. Forexample, the 2D Fourier transform can be calculated (e.g., using a 2DFFT algorithm, if the sub-sampling window width and length are multiplesof 2) and the magnitude data stored in a matrix, wherein the DCfrequency component is represented by the origin of the axis in thefrequency domain. FIG. 5 is a schematic illustration of a sub-sampledROI for which a 2D Fourier transform is calculated. The 2D Fouriertransform data set can then be divided into four concentric regions 502,504, 506, and 508 based on frequency content. The outermost region 502,for example, represents a portion of the sample image having the highestspatial frequency content.

The magnitudes of the spatial frequencies in each of regions 502, 504,506, and 508 can be integrated and normalized to the total signalmagnitude. The integrated magnitudes form the next four elements in theROI's feature vector, and each corresponds to a percentage of Fouriertransform signal within a certain range of spatial frequencies.

In general, in embodiments, the spatial Fourier transform data can bepartitioned into any number of selected frequency regions (subject tothe spatial Nyquist limit) and the integrated intensities from theseregions correspond to textural features of the image. Some or all ofthese textural features can be incorporated into the feature vector forthe ROI.

The remaining two calculations in this present example of determiningthe feature vector can be derived from first order pixel statistics. Forexample, the ninth and tenth calculations can correspond to the mean andstandard deviation of the pixel values within the ROI. In general, otherstatistical measures can also be useful as feature vector elements.These quantities can be derived from first order or higher orderstatistical measures (e.g., the variance in pixel values, which isderived from the second moment of the statistical distribution of pixelvalues).

Referring again to FIG. 3, following calculation of each of the elementsof the feature vector that corresponds to the currently selected class,the feature vector is stored. A logical decision 314 follows next. Iffeature vectors for all of the N_(c) classes have been calculated andstored, then subsequent neural network training steps, beginning withstep 318, are taken. Conversely, if feature vectors have not beencalculated, then in step 316 a class indicator i is incremented, whichis equivalent to selecting a new class and its associated ROI, and theanalysis for the newly selected class begins at step 308.

When all of the feature vectors for the identified classes have beencalculated, the next step in the sequence is step 318, which includesselecting a sequence of the calculated feature vectors for use astraining vectors corresponding to the N_(c) classes identified in step302. The set of training vectors can include multiple vectorscorresponding to different ROIs for each of the classes. However, careis taken to ensure that each identified class contributes the samenumber of distinct training vectors to the training set in step 318.This balancing of the relative abundance of different training vectorsin the training set is important in order to ensure that the neuralnetwork is trained in unbiased fashion with respect to the differentclasses in sample images.

In step 320, the set of training vectors is submitted to the neuralnetwork-based classifier for classification. The vectors are classifiedone-by-one in random order, and for each one the neural network developsan output estimate of what class the vector belongs to, and this iscompared against the actual known class corresponding to the vector. Thedifference between the network output and the actual class is termed theerror. The network is then adjusted using a method such as gradientdescent back-propagation, or other error-adjustment techniques, whichacts to adjust the network values and produce a reduced error value.When all of the training ROIs have been assigned by the network, theclassification accuracy can be determined in step 322, either manuallyby an operator or automatically by calculating a score that indicates,for example, what percentage of ROIs were classified correctly.

Logical step 324 includes a decision based on the accuracy of theclassification of the training ROIs. If the accuracy is higher than aselected threshold (which, in some embodiments, can be set to 100%accuracy, for example) then the neural network is considered to havebeen suitably trained and the training sequence finishes in step 326.However, if the accuracy falls below the selected threshold, then thesteps involving classification of training ROIs are repeated. That is,training vectors are prepared as in step 318 and test classification ofthese ROIs by the neural network begins again. The vectors may be thesame set used in the initial training, or may be a different set ofvectors. Repeatedly training on a single set is productive as long asthe error network adjustment continues to improve classificationaccuracy. In many embodiments, 100% accuracy is achieved on a first setof training ROIs. However, the threshold for successful training may beset lower than 100 percent if that is desirable. This may occur if onedoes not have perfect knowledge of class identity for the training ROIs,or if the samples themselves are highly variable and a wide range oftraining ROIs are employed.

Optimizing the Neural Network

Following successful training of the neural network-based classifier,the network can optionally be optimized with respect to the number offeatures used to classify sample images. Optimizing the network in thismanner can increase the efficiency and speed of classificationoperations.

FIG. 6 is a flow chart 600 that includes an optional series of stepsinvolved in optimizing a trained neural network. First step 602 includesgenerating a random sequence of training vectors to test the performanceof the neural network. As before, the sequence of training vectors isconstructed such that there exists an equal number of vectorscorresponding to each of the N_(c) classes identified previously.

Step 604 includes choosing the number of neural network classificationfeatures N_(ƒ). Initially, the value of N_(ƒ) typically consists of allthe features that were calculated, for all image planes, which is thenumber of elements in the feature vector. Subsequent iterations of theoptimization sequence can reduce the value of N_(ƒ) according to theclassification performance of the neural network.

In step 606, the random sequence of vectors generated in step 602 issubmitted to the neural network for classification. The classificationof individual vectors is performed by the trained network in a mannerconsistent with the prior discussion. A feature vector is calculated foreach ROI (e.g., based on one or more sub-sampled windows in the ROI),and the ROI is assigned to a particular class according to the knownfeature vectors for the various identified classes. In step 608, aclassification accuracy score is determined either by visual inspection(e.g., by an operator) or by calculating the fraction of correctclassification results.

In order to assess the relative significance of each of the N_(ƒ)features to the performance of the neural network, the mean featurevalue μ_(j) for each of the j classification features is calculated instep 610. Calculation of a mean feature value can be accomplished, forexample, by calculating a mean value of the elements in a feature vectorcorresponding to a particular class. The elements in the feature vectorcan be weighted equally or differently in performing the calculation ofμ_(j).

In a further step 612, the weighted contribution W_(j) of each feature jof the N_(ƒ) total features under consideration by the neural network iscalculated according to

$\begin{matrix}{W_{j} = {\sum\limits_{k = 1}^{N_{f}}{\mu_{j}\gamma_{k}}}} & (12)\end{matrix}$where the γ_(k) values are the node-to-node coupling constants withinthe neural network. Using Equation (12), the weighted contributions ofeach of the features (which generally correspond to classes) can beevaluated. In step 614, classification feature s having the smallestweighted contribution W_(s) is identified as the “weakest”classification feature and removed from the set of classificationfeatures considered by the neural network.

In steps 616 and 618, a new random sequence of training vectors isgenerated according to the procedures discussed previously, and thetraining vectors are classified by the modified neural network, whichnow includes one less feature. A classification accuracy score isdetermined following classification of the vectors.

In logic step 620, the classification accuracy score is compared againsta selected accuracy threshold. If the accuracy score is higher than thethreshold, then the removed feature is deemed to be insignificant enoughthat it can be permanently removed from consideration by the neuralnetwork. The number of neural network classification features N_(ƒ) isreduced by one in step 622 and logical flow returns to step 610, wherenew mean feature values are calculated for the newly reduced set ofclassification features in the neural network. In some embodiments,before logical flow returns to step 610, the neural network can beretrained in order to adapt to the smaller number of features. This stepis not necessary, but may be employed in some embodiments to improve theaccuracy and/or speed of classification.

If the accuracy score is lower than the selected threshold, the removedfeature s is deemed to have been significant after all, and isre-introduced into the neural network in step 624. This completes theoptimization of the network in step 626, and the network is then readyfor use in classifying samples based on image sets.

If all features corresponding to a given input image plane are removedduring the optimization process, that input plane is superfluous andneed not be acquired in order to provide the classification signal.Further improvement in efficiency can be obtained by not acquiring suchplanes in future measurements, if the plane is not required for otherpurposes. The determination of which image planes are necessary can bemade once, when devising a measurement protocol; or, it may be madeand/or reviewed on an ongoing basis over time, in settings where factorssuch as sample variability may lead to changes in what image planes arenecessary or helpful in making a classification.

Classification Using a Trained Neural Network

The procedure by which a sample is classified according to its imagestack is shown in FIG. 7. The figure includes a flow chart 700 thatillustrates a series of steps in the classification procedure. In step702, a particular image stack for the sample is chosen forclassification, and in step 704, a number of regions N_(r) within theimage stack are selected for analysis. In some embodiments, the regionsselected are subsets of the entire image. In other embodiments, theentire image can be selected for analysis.

The image stack being selected for analysis may include one or moreimages. The images in the image stack may include one or more rawspectral images, one or more composite images, and/or one or moreunmixed images. For example, in certain embodiments, the image stack mayinclude one composite image to provide spectral and spatial informationand one gray scale image to provide spatial information. In otherembodiments, for example, the image stack may include a set of unmixedimages. Furthermore, in some embodiments, for example, theclassification may be applied to only a single image containing onlyspatial (and no spectral) information. In any case, the neural networkis trained in anticipation of the type of image stack being selected.

In step 706, a length l and width w of a sub-sampling window areselected. As discussed previously, the length and width of thesub-sampling window are typically chosen to be smaller than the meanlength and width of each of the N_(r) regions selected for analysis. Inaddition, step 706 includes selection of window offset increments Δl andΔw. The offset increments are used to translate the sub-sampling windowover the classification regions of the sample image in order to ensurethat each of the pixels within the regions is classified at least once.In some embodiments, the values of Δl and Δw are both chosen to besmaller than l and w, respectively, so that at least some pixels areclassified multiple times since each translation of the sub-samplingwindow to a new position leaves a fraction of the previous window'spixels within the new window.

In step 708, one of the regions selected in step 704 is submitted to thetrained (and optionally, optimized) neural network for classification.The classification of the pixels in the windowed region is performed instep 710. The classification procedure is an iterative one, in whichpixels within the selected region can be provisionally assigned aclassification multiple times. The procedure begins by positioningsub-sampling window 802 within the selected region 800, as shown in FIG.8. Sub-sampling window 802 has a length l in the x direction and a widthw in the y direction. The offset increments Δl and Δw are smaller thanthe length and width of the sub-sampling window, respectively.

In the first position of the sub-sampling window, each of the imagepixels within the window is assigned a provisional classification basedon the classification of the overall window region by the neural networkusing the methods discussed previously. The provisional pixelclassifications can be stored within a pixel histogram for futurereference. This corresponds to step 710. Referring again to FIG. 7, thenext step is a logical decision 712 based on whether sub-sampling of theregion is complete. If sub-sampling of the region is not complete, thesub-sampling window is then translated in the x and y directions byincrements Δl and Δw, respectively, as shown in step 714. The imagepixels that fall within the new sub-sampling window position are thenclassified as before in step 710.

The procedure is illustrated schematically in the lower part of FIG. 8,in which window 802 a represents the first position of the sub-samplingwindow and window 802 b represents the second position of the windowfollowing translation. The classification of the pixels within thesecond window 802 b by the neural network then proceeds as before. Notethat the pixels that fall within shaded region 804 are classified asecond time, since they are positioned within both windows 802 a and 802b. The multiple classification of image pixels is a particular featureof certain embodiments of the methods disclosed herein.

Returning again to step 710 in FIG. 7, the classifications of individualpixels are again stored in the pixel histogram, and then thesub-sampling window is again translated and the classification procedurebegins anew for a new window position. This iterative procedure,consisting of steps 710 through 714, can be specified to repeat for aselected number of window translations, such that a pixel classificationhistogram is built up.

Note that while FIG. 8 depicts the translation associated with step 714as having both increments Δl and Δw, this is not necessary. For example,in some embodiments, the translation may scan horizontally, followed bya vertical translation when each horizontal scan across the selectedregion is complete, or vice versa. In certain embodiments, for example,each translation will correspond to a step of a single pixel and thetranslations across the region will generally result in each pixel beingclassified by number of pixels in the sampling window. Furthermore, inother embodiments, the translations need not be sequential. For example,the window translations can be systematic or random within the selectedimage region, although in some embodiments, an additional constraintthat must be satisfied prior to termination of the classificationprocedure stipulates that all pixels within the selected region shouldbe classified at least once, and preferably multiple times. Such aconstraint is optional, however, and need not be imposed. Once thesub-sampling of the selected image region is complete, logical decision716 determines a course of action based upon whether all of the selectedregions of the sample image stack have been provisionally classified(e.g., a histogram of provisional classifications has been developed forevery pixel in the selected regions of the selected image stack). Ifthere are remaining unclassified regions, then counter i is incrementedin step 718 (equivalent to selecting one of the unclassified regions)and classification of the selected regions begins at step 708 of flowchart 700.

Alternatively, if each of the regions of the sample image have beenprovisionally classified, then the initial classification procedure isfinished and control passes to step 722, in which a final pixelclassification step is performed based on the accumulated histogram datafor each of the pixels. Due to the fact that pixels can be classifiedmultiple times, entries in the classification histogram for particularpixels may not all be the same, and a pixel can be provisionallyclassified into more than one class.

A wide variety of algorithms can be used to establish a classificationfor a particular pixel from the histogram data. For example, the finalclassification of a given pixel can be the class to which the pixel wasmost frequently assigned. Alternatively, a more complex analysis of thestatistical information in the histogram can be used to assign the finalclassification. For example, a pixel's classification can be establishedas the mean, median, or mode of the distribution of classifications forthat pixel. Alternatively, more advanced statistical methods such asfuzzy logic or Bayesian logic can be applied to the histogram data todetermine classifications for each of the image pixels.

In some embodiments, the histogram data can be used to “flag” particularregions of the sample according to classification. For example, if thehistogram data for a particular pixel includes even a single instance inwhich the pixel was classified as belonging to a particular class, stepscan be taken to ensure that the pixel is positively identified. Warningmessages or sounds can be produced, or a sample image having theidentified pixels highlighted for easy identification can be displayed.Flagging techniques can be particularly useful when tissue samples areexamined for the presence of harmful agents or structures such aspathogens and cancer cells.

The final step 724 includes generating a classification map for thesample based on the final classification of step 722, or more generally,on the provisional pixel classification histogram data generated in theearlier steps. The classification map can include, for example, an imageof the sample with classified regions highlighted in order to enhancecontrast. The map can include, in some embodiments, multiple images ofthe sample, where each image only those portions of the sample thatbelong to a particular class, as identified by the neural network. Theclassification map can also include numerical data specifying classifiedsample regions, and statistical information such as the distribution andrelative abundance of various classes within the sample. Thisinformation is particularly useful when the classified regionscorrespond to different structural, chemical, or biological entitieswithin the sample. The classification image map can be displayed ondisplay device 118, for example, and can be stored in electronic form ona storage medium by electronic control system 114. Generation of theclassification map completes the classification procedure, and generallyyields accurate class data for a wide variety of samples.

Optical System Components

System 100 can include a wide variety of optical elements and devicesfor capturing images of a sample that are used in subsequentclassification algorithms. Light source 102 can be an incoherent lightsource such as an incandescent lamp, a fluorescent lamp, or a diode.Light source 102 can also be a coherent source such as a laser source,and the coherent source can provide continuous wave (CW) or pulsedlight. Light source 102 may contain multiple light source elements forproducing light having a range of wavelengths (e.g., multiple diodes).When the light produced by light source 102 is pulsed (i.e.,time-gated), various properties of the light pulses can be manipulatedaccording to control signals provided to light source 102 fromelectronic control system 114 via communication line 132. Light source102 can also include various optical elements such as lenses, mirrors,waveplates, and nonlinear crystals, all of which can be used to producelight having selected characteristics. In general, light source 102includes optical elements and devices configured to provide light havingdesired spectral, spatial, and, in some embodiments, temporalproperties.

Light conditioning optics 104 and light collecting optics 110 caninclude a variety of optical elements for manipulating the properties oflight incident on, and emitted from, a sample of interest. For example,light conditioning optics 104 and light collecting optics 110 can eachinclude spectral filter elements for selecting particular wavelengthbands from incident and emitted light. The spectral filter elements caninclude, for example, interference filters mounted on a filter. In someembodiments, adjustable filter elements based on liquid crystal maskscan be used to change the spectral properties of the incident or emittedlight. Liquid crystal based devices can be controlled by electroniccontrol system 114 via communication lines 134 and 138.

Light conditioning optics 104 and light collecting optics 110 can alsoinclude elements such as spatial light masks, spatial light modulators,and optical pulse shapers in order to manipulate the spatialdistribution of light incident on, or emitted from, a sample. Spatiallight modulators and other adaptive devices can also be controlled viacommunication lines 134 and 138 by electronic control system 114.

Finally, light conditioning optics 104 and light collecting optics 110can include other common optical elements such as mirrors, lenses,beamsplitters, waveplates, and the like, configured in order to impartselected characteristics to the incident or emitted light.

In general, detector 112 includes one or more measurement devicesconfigured to detect and capture light emitted by a sample as multipleimages of the sample. Detector 112 can include devices such as CCDarrays and photomultiplier tubes, along with their respective controlsystems, for acquiring the images. The adaptive optical devices indetector 112 can, in general, be controlled by electronic control system114 via communication line 130.

Software

The steps described above in connection with various methods forcollecting, processing, analyzing, interpreting, and displayinginformation from samples can be implemented in computer programs usingstandard programming techniques. Such programs are designed to executeon programmable computers or specifically designed integrated circuits,each comprising an electronic processor, a data storage system(including memory and/or storage elements), at least one input device,and least one output device, such as a display or printer. The programcode is applied to input data (e.g., images from the detector) toperform the functions described herein and generate output information(e.g., images showing classified regions of samples, statisticalinformation about sample components, etc.), which is applied to one ormore output devices. Each such computer program can be implemented in ahigh-level procedural or object-oriented programming language, or anassembly or machine language. Furthermore, the language can be acompiled or interpreted language. Each such computer program can bestored on a computer readable storage medium (e.g., CD ROM or magneticdiskette) that when read by a computer can cause the processor in thecomputer to perform the analysis and control functions described herein.

EXAMPLES

The following examples are intended to be exemplary of the systems andmethods disclosed herein, but should not in any way be construed aslimiting the scope of the subsequent claims.

FIG. 10A shows an example of a sample of rat blood that is classifiedaccording to some of the methods of the present disclosure. The bloodsample includes 4 classes: background 1002, red cells 1004, monocytes1006, and polymorphonuclear neutrophils (PMNs) 1008. A set of spectralimages corresponding to incident light transmitted through the samplewere collected and then transformed from measured intensities to opticaldensities (ODs). The resulting transformed images formed a spectral cubeof image data.

The spectral image cube was unmixed into separate images correspondingto a red component 1010 and a blue component 1012 of the blood sample,as shown in FIG. 10B. FIG. 10C and FIG. 10D show the results of thisspectral unmixing operation. FIG. 10C shows an example of an unmixedimage corresponding to the red component 1010 and FIG. 10D shows anexample of an unmixed image corresponding to the blue component 1012.

Following the unmixing step, a composite plane was generated by a linearramp function used as the weighting function, so that the unmixed planesand composite plane formed a 3-plane stack. Next, training regions wereselected on the image stack, and a neural network-based classifier wastrained according to the selected regions. FIG. 10E shows selectedtraining regions superimposed on an image of the sample. Training of theneural network includes calculation of features related to theidentified training regions. An expanded view of this process for thetraining regions is shown in FIG. 10F. The left side of FIG. 10F shows aview of an expanded region of a sample image that includes selectedtraining regions. On the right side of FIG. 10F, the selected trainingregions have been sub-sampled, and the sub-sampling windows aresuperimposed over the regions.

The trained neural network-based classifier was then used to classifythe remaining regions of the images. The results are shown in FIG. 10G.The image features corresponding to the background class 1002, red cellclass 1004, monocyte class 1006, and PMN class 1008 are all accuratelydetermined and identified using the neural network-based classifier.

In another example, a 3-plane RGB image was generated from the sameimage cube of spectral images, and selected regions of the RGB imagewere used to train and optimize a neural network. This RGB image wasgenerated by summing all the spectral bands in the blue to form a blueplane, summing all the spectral bands in the green to form a greenplane, and summing all the spectral bands in the red to form a redplane. The result mimics what would have resulted if the scene wereimaged using a conventional RGB camera. The trained and optimized neuralnetwork was then used to classify the remainder of the composite image.FIG. 10H shows the RGB image, and FIG. 10I shows the results of theclassification operations carried out. The image features correspondingto the background class 1002, red cell class 1004, monocyte class 1006,and PMN class 1008 are all accurately determined and identified usingthe neural network-based classifier.

The automated methods disclosed herein provide an effective means forclassifying the blood sample.

A number of embodiments of the invention have been described.Nevertheless, it will be understood that various modifications may bemade without departing from the spirit and scope of the invention.Accordingly, other embodiments are within the scope of the followingclaims.

What is claimed is:
 1. A method for assessing a biological sample, themethod comprising using one or more electronic processors to perform thefollowing steps: decomposing a set of n spectral images of the sampleinto a set of m unmixed images, wherein each member of the unmixed imageset corresponds to a spectral contribution from a different component inthe sample, and wherein n≧4; and classifying spatial locations in thesample corresponding to pixels in the set of unmixed images based on animage stack comprising one or more members of the set of unmixed imagesto assess whether each spatial location corresponds to one or more of pclasses.
 2. The method of claim 1, wherein the decomposing comprisesdecomposing the set of n spectral images into a set of m≦3 unmixedimages.
 3. The method of claim 1, wherein the decomposing comprisesdecomposing the set of n spectral images into a set of m<n unmixedimages.
 4. The method of claim 1, wherein the set of spectral imagescomprises n=4 images and the set of unmixed images comprises m=3 images.5. The method of claim 1, wherein the classifying comprises assessingwhether each spatial location corresponds to one or more of p=m classes.6. The method of claim 1, wherein the classifying comprises assessingwhether each spatial location corresponds to one or more of p<m classes.7. The method of claim 1, wherein p>1.
 8. The method of claim 1, whereinp=1.
 9. The method of claim 1, wherein one of the p classes is a diseasestate.
 10. The method of claim 1, wherein one of the p classes is a cellor tissue type.
 11. The method of claim 1, wherein one of the p classesis a biological structure type.
 12. The method of claim 1, wherein thebiological sample comprises cells.
 13. The method of claim 1, whereinthe decomposing further comprises determining amounts of at least someof the sample components in at least some of the spatial locations ofthe sample.
 14. The method of claim 13, wherein the decomposing furthercomprises determining amounts of each of the sample components in atleast some of the spatial locations of the sample.
 15. The method ofclaim 1, wherein the classifying comprises classifying at least some ofthe spatial locations based on information derived from pixels in theimage stack neighboring the pixels to which the spatial locationscorrespond.
 16. The method of claim 15, wherein the classifyingcomprises classifying the at least some of the spatial locations basedon both spectral and spatial information derived from the neighboringpixels.
 17. The method of claim 1, further comprising obtaining the setof spectral images by spectrally filtering illumination light directedto the sample to select light in each of a plurality of differentwavelength bands, wherein each member of the set of spectral imagescorresponds to a different one of the wavelength bands.
 18. The methodof claim 1, further comprising obtaining the set of spectral images byspectrally filtering light received from the sample in each of aplurality of different wavelength bands, wherein each member of the setof spectral images corresponds to a different one of the wavelengthbands.
 19. The method of claim 1, wherein classifying spatial locationscorresponding to pixels in the set of unmixed images comprises:positioning a sampling window within the image stack to select a portionof the image stack for classification, the selected portion comprisingmultiple pixels; classifying the selected portion to assess whether theselected portion corresponds to the one or more of p classes, whereineach of the multiple pixels in the selected portion is provisionallyassessed as having the same classification with respect to the one ormore of p classes as the selected portion; translating the samplingwindow to select a second portion of the image stack comprising multiplepixels for classification and classifying the second portion to assesswhether the second portion corresponds to the one or more of p classes,wherein each of the multiple pixels in the second portion isprovisionally assessed as having the same classification with respect tothe one or more of p classes as the second portion; repeating thetranslating and classifying for additional portions of the image stackuntil at least some of the pixels in the image stack have beenprovisionally assessed multiple times as part of different portionsselected by the sampling window; and assessing whether each of at leastsome of the spatial locations corresponding to the pixels that have beenprovisionally assessed multiple times corresponds to the one or more ofp classes based on the multiple provisional assessments.
 20. The methodof claim 1, further comprising using a machine learning classifier toclassify the spatial locations, wherein the machine learning classifiercomprises at least one member selected from the group consisting of aneural network-based classifier, a genetic algorithm-based classifier,and a support vector machine-based classifier.
 21. A system forassessing a biological sample, the system comprising: a light source;light conditioning optics positioned to direct light from the source tothe sample; light collecting optics positioned to direct light from thesample to a detector; a detector configured to receive light from thelight collecting optics and to record a set of n spectral images of thesample based on the received light, wherein n≧4; and an electronicprocessor connected to the detector and configured to: decompose the setof n spectral images into a set of m unmixed images, wherein each memberof the unmixed image set corresponds to a spectral contribution from adifferent component in the sample; and classify spatial locations in thesample corresponding to pixels in the set of unmixed images based on animage stack comprising one or more members of the set of unmixed imagesto assess whether each spatial location corresponds to one or more of pclasses.
 22. The system of claim 21, wherein the electronic processor isconfigured to decompose the set of n spectral images into a set of m≦3unmixed images.
 23. The system of claim 21, wherein the electronicprocessor is configured to decompose the set of n spectral images into aset of m<n unmixed images.
 24. The system of claim 21, wherein thedetector is configured to record a set of n=4 spectral images of thesample, and wherein the electronic processor is configured to decomposethe set of n=4 spectral images into a set of m=3 unmixed images.
 25. Thesystem of claim 21, wherein the electronic processor is configured toassess whether each spatial location corresponds to one or more of p≦mclasses.
 26. The system of claim 21, wherein the electronic processor isconfigured to assess whether each spatial location corresponds to one ormore of p>1 classes.
 27. The system of claim 26, wherein one of the pclasses is a disease state.
 28. The system of claim 26, wherein one ofthe p classes is a cell or tissue type.
 29. The system of claim 26,wherein one of the p classes is a biological structure type.
 30. Thesystem of claim 21, wherein the electronic processor is furtherconfigured to determine amounts of at least some of the samplecomponents in at least some of the spatial locations of the sample.